Fusing deep learned and hand-crafted features of appearance, shape, and dynamics for automatic pain estimation

Automatic continuous time, continuous value assessment of a patient's pain from face video is highly sought after by the medical profession. Despite the recent advances in deep learning that attain impressive results in many domains, pain estimation risks not being able to benefit from this due to the difficulty in obtaining data sets of considerable size. In this work we propose a combination of hand-crafted and deep-learned features that makes the most of deep learning techniques in small sample settings. Encoding shape, appearance, and dynamics, our method significantly outperforms the current state of the art, attaining a RMSE error of less than 1 point on a 16-level pain scale, whilst simultaneously scoring a 67.3% Pearson correlation coefficient between our predicted pain level time series and the ground truth

Unsupervised learning of facial expression components

The face is one of the most important means of non-verbal communication. A lot of information can be gotten about the emotional state of a person just by merely observing their facial expression. This is relatively easy in face to face communication but not so in human computer interaction. Supervised learning has been widely used by researchers to train machines to recognise facial expressions just like humans. However, supervised learning has significant limitations one of which is the fact that it makes use of 'labelled' facial images to train models to identify facial actions. It is very expensive and time consuming to label face images. It takes about an hour to label a 5min video. In addition, more than 7000 distinct facial expressions can be created from a combination of different facial muscle actions [72]. The amount of labelled face images available for facial expression analysis is limited in supply and do not cover all the possible facial expressions. On the other hand, it is quite easy to collect or record a large amount of raw unlabelled data of people communicating without incurring so much cost. This research therefore used an unsupervised learning to identify facial action units. Similar appearance changes were identified from a large data set of face images without knowing beforehand which facial expressions they belong to and then the identified patterns or learned features were assigned to distinct facial expressions. For the purpose of comparison with the supervised learning approach, the learned features were used to train models to identify facial actions. Results showed that in 70% of the cases, the accuracy from the unsupervised learning model surpassed the performance of the supervised learning model

Automatic pain assessment from face video (continuous pain intensity estimation in adults and newborns)

Pain assessment is a very crucial aspect of medical diagnosis as it is symptomatic of many medical conditions. In cases where a patient is unable to self-report pain, pain assessment is done by a clinician via observation of behavioural changes and vital signs. However, this method is highly subjective and discontinuous in time. In order to introduce an objective measure to clinical pain assessment and support real-time pain monitoring, automatic pain recognition models have been proposed but the performance of these models is still limited by the imbalanced and small sample pain data available for training. In addition, there is currently a dearth of information on the usability and impact of such tools in clinical settings. This thesis aims to develop novel computer vision and machine learning techniques that can achieve good pain estimation on small and sparse pain datasets and also explore the applicability of automated pain assessment tools to clinical settings. Regarding the problem of insufficient data for automatic pain recognition, this thesis presents and describes the collection of a geographically diverse multimodal newborn and infant pain dataset containing over 200 participant videos with framewise pain intensity annotations. Furthermore, to address the problem of learning from small pain data with sparse representation for higher pain levels, two novel methods of learning discriminative pain features for pain intensity estimation are proposed - a Hybrid Deep-learned and Hand-crafted (HDH) feature framework and a Cumulative Attribute (CA) learning framework. Evaluation of the HDH feature model on the UNBC McMaster pain dataset yielded state-of-the-art performance with an RMSE of 0.99 and a Pearson Correlation of 0.67. Similarly, an analysis of the CA learning framework revealed that models trained on the CA features consistently outperformed those trained on the corresponding low-level features and the performance improvement was due to the CA feature's ability to make better predictions for sparse higher pain classes. Additional evaluation on the newborn pain data resulted in a performance comparable to human error. As a step towards investigating the suitability of computer-assisted pain assessment tools in clinical contexts, a user study was conducted with clinicians from a Neonatal Intensive Care Unit (NICU). Using the aforementioned models as early prototypes, qualitative and quantitative methods were employed to gauge acceptance and identify usability design issues. Findings from the study generated useful NICU environment-specific and context design issues which researchers should consider when developing video-based pain assessment tools. Preliminary experimental results from the proposed models, as well as the insights garnered from the stakeholder survey, could potentially lead to improved clinical pain management. Likewise, the dataset presented would promote research in automatic newborn pain assessment and serve as a benchmarking platform for future methods

Sensitive Pictures:Emotional Interpretation in the Museum

Museums are interested in designing emotional visitor experiences to complement traditional interpretations. HCI is interested in the relationship between Affective Computing and Affective Interaction. We describe Sensitive Pictures, an emotional visitor experience co-created with the Munch art museum. Visitors choose emotions, locate associated paintings in the museum, experience an emotional story while viewing them, and self-report their response. A subsequent interview with a portrayal of the artist employs computer vision to estimate emotional responses from facial expressions. Visitors are given a souvenir postcard visualizing their emotional data. A study of 132 members of the public (39 interviewed) illuminates key themes: designing emotional provocations; capturing emotional responses; engaging visitors with their data; a tendency for them to align their views with the system's interpretation; and integrating these elements into emotional trajectories. We consider how Affective Computing can hold up a mirror to our emotions during Affective Interaction.Comment: Accepted for publication in CHI 202

ALTCAI: Enabling the Use of Embodied Conversational Agents to Deliver Informal Health Advice during Wizard of Oz Studies

We present ALTCAI, a Wizard of Oz Embodied Conversational Agent that has been developed to explore the use of interactive agents as an effective and engaging tool for delivering health and well-being advice to expectant and nursing mothers in Nigeria. This paper briefly describes the motivation and context for its creation, ALTCAI’s various components, and presents a discussion on its adaptability and potential uses in other contexts, as well as on potential future work on extending its functionality

Designing an adaptive embodied conversational agent for health literacy

ccess to healthcare advice is crucial to promote healthy societies. Many factors shape how access might be constrained, such as economic status, education or, as the COVID-19 pandemic has shown, remote consultations with health practitioners. Our work focuses on providing pre/post-natal advice to maternal women. A salient factor of our work concerns the design and deployment of embodied conversation agents (ECAs) which can sense the (health) literacy of users and adapt to scaffold user engagement in this setting. We present an account of a Wizard of Oz user study of 'ALTCAI', an ECA with three modes of interaction (i.e., adaptive speech and text, adaptive ECA, and non-adaptive ECA). We compare reported engagement with these modes from 44 maternal women who have differing levels of literacy. The study shows that a combination of embodiment and adaptivity scaffolds reported engagement, but matters of health-literacy and language introduce nuanced considerations for the design of ECAs

Design and evaluation of virtual human mediated tasks for assessment of depression and anxiety

Virtual human technologies are now being widely explored as therapy tools for mental health disorders including depression and anxiety. These technologies leverage the ability of the virtual agents to engage in naturalistic social interactions with a user to elicit behavioural expressions which are indicative of depression and anxiety. Research efforts have focused on optimising the human-like expressive capabilities of the virtual human, but less attention has been given to investigating the effect of virtual human mediation on the expressivity of the user. In addition, it is still not clear what an optimal task is or what task characteristics are likely to sustain long term user engagement. To this end, this paper describes the design and evaluation of virtual human-mediated tasks in a user study of 56 participants. Half the participants complete tasks guided by a virtual human, while the other half are guided by text on screen. Self-reported PHQ9 scores, biosignals and participants' ratings of tasks are collected. Findings show that virtual-human mediation influences behavioural expressiveness and this observation differs for different depression severity levels. It further shows that virtual human mediation improves users' disposition towards tasks

An Assessment of Final Year Pre-Service Teachers’ Readiness to Use ICT to Teach: Implication for COVID-19 Education Response

Readiness of teachers to use Information and Communication Technology (ICT) to teach is currently a significant issue due to its relationship to online teaching/learning which is a global COVID-19 education response. Preparing future teachers to ICT effectively to facilitate variety of online learning modes is a huge challenge for teacher training institutions, especially in developing nations like Nigeria. In this study, the ICT readiness of final year pre-service teachers in the degree programme of College of Education, Agbor, Delta State, Nigeria was assessed. The data gathered using a Likert questionnaire was computer-analyzed. The findings suggest that the pre-service teachers were ready with respect to the following indicators.· Awareness and motivation to use ICT to teach.· Confidence to use ICT to teach.· Availability of internet/mobile network facilities in their environments.· They were very ready with respect to perception on the benefits of ICT in teaching and learning.They did not express readiness in terms of the following indicators.*Acquisition of adequate knowledge and skills on the use of ICT devices during training.*Personal ownership of ICT devices (e.g., computers/laptops).It is therefore recommended that compulsory personal ownership of laptops and relevant ICT devices by pre-service teachers could help them practice and acquire necessary ICT skills more adequately during their training

Unsupervised learning of facial expression components

The face is one of the most important means of non-verbal communication. A lot of information can be gotten about the emotional state of a person just by merely observing their facial expression. This is relatively easy in face to face communication but not so in human computer interaction. Supervised learning has been widely used by researchers to train machines to recognise facial expressions just like humans. However, supervised learning has significant limitations one of which is the fact that it makes use of 'labelled' facial images to train models to identify facial actions. It is very expensive and time consuming to label face images. It takes about an hour to label a 5min video. In addition, more than 7000 distinct facial expressions can be created from a combination of different facial muscle actions [72]. The amount of labelled face images available for facial expression analysis is limited in supply and do not cover all the possible facial expressions. On the other hand, it is quite easy to collect or record a large amount of raw unlabelled data of people communicating without incurring so much cost. This research therefore used an unsupervised learning to identify facial action units. Similar appearance changes were identified from a large data set of face images without knowing beforehand which facial expressions they belong to and then the identified patterns or learned features were assigned to distinct facial expressions. For the purpose of comparison with the supervised learning approach, the learned features were used to train models to identify facial actions. Results showed that in 70% of the cases, the accuracy from the unsupervised learning model surpassed the performance of the supervised learning model